Bayesian Linear Combination of Neural Networks

Classifier ensembles have been one of the main topics of interest in the neural networks, machine learning and pattern recognition communities during the past fifteen years [21,28,16,17,26,36,27,23,11]. They are currently one of the state of the art techniques available for the design of classification systems and an effective option to the traditional approach based on the design of a single, monolithic classifier in many applications. Broadly speaking, two main choices have to be made in the design of a classifier ensemble: how to generate individual classifiers and how to combine them. Two main approaches have emerged to deal with these design steps: coverage optimisation, focused on generating an ensemble of classifiers as much complementary as possible, which are then fused with simple combining rules, and decision optimisation, focused on finding the most effective combining rule to exploit at best a given classifier ensemble [21]. One of the most studied and widely used combining rules, especially in the former approach, is the linear combination of classifier outputs. Linear combiners are often used for neural network ensembles, given that neural networks provide continuous outputs. The simplicity of linear combiners and their continuous nature favoured the development of analytical models for the analysis of the performance of ensembles of predictors, both for the case of regression problems and for the relatively more complex case of classification problems. In this chapter, we give an overview on ensembles of linearly combined neural networks. Our survey is focused on a Bayesian analytical model introduced about ten years ago in works by K. Tumer and J. Ghosh [31,32] and recently extended by the authors [8,4]. Basically, this model allows to quantify the advantage attainable by linearly combining an ensemble of classifiers, in terms of the reduction in misclassification probability. Although based on strict assumptions to make it analytically tractable, this model allows to point out the main factors which affect the performance of linearly combined classifier ensembles and suggests simple guidelines for their design. It was also recently exploited to develop a novel method for training ensembles of linearly combined neural networks [37] and to analyse the behaviour of bagging (a well known technique for constructing classifier ensembles) as a function of the ensemble size [9]. This chapter starts with an overview of past works on ensembles of linearly combined neural networks, both for regression and classification problems (section 2). The analytical model by Tumer and Ghosh is then presented, followed by the extension given by the authors, and its main results and implications are discussed in section 3. Finally, some experimental results are reported in section 4 to illustrate the main results of this model.